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Executive  Summary 


All  U.S.  Marines  must  qualify  as  marksman;  as  a  result  the  training  demand  on  live  fire  rifle  ranges  is 
significant.  Marksmanship  simulators  such  as  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  may 
help  relieve  some  of  this  pressure  on  traditional  ranges,  but  the  training  effectiveness  of  such  systems 
must  be  evaluated  and  verified.  The  goal  of  the  current  effort  was  to  develop  a  subjective  survey  for 
evaluating  marksmanship  training  systems,  as  a  possible  alternative  to  lengthy  and  resource-demanding 
training  effectiveness  evaluations.  A  task  analytic  approach  was  used  to  break  down  the  marksmanship 
domain,  as  presented  in  the  USMC  Rifle  Marksmanship  Manual,  into  sub-tasks  that  were  converted  into 
training-task  statements  which  were  arranged  into  a  survey  format.  The  survey  asked  USMC 
marksmanship  instructors  to  rate  each  task  statement  on  Importance,  Difficulty,  Known  Distance  (KD) 
Range  Training  Effectiveness,  and  ISMTTraining  Effectiveness.  Surveys  were  distributed  to  39 
instructors  and  22  of  these  Subject  Matter  Experts  (SMEs)  returned  completed  surveys. 

The  interrater  agreement  for  the  survey  was  generally  good,  which  is  a  basic  requirement  for  this  tool  to 
have  utility  in  assessing  marksmanship  trainers.  Importance  ratings  were  high,  with  a  mean  of  4.3  (all  of 
the  scales  on  the  survey  ranged  from  1  to  5).  This  was  not  a  surprising  result  since  the  task  statements 
were  generated  from  the  USMC  Rifle  Marksmanship  Training  Manual.  The  mean  Difficulty  rating  was 
2.8,  near  the  scale  midpoint,  and  respondents  used  a  fairly  wide  range  of  the  available  scale  indicating 
that  the  SMEs  were  able  to  successfully  rate  tasks  relative  to  each  other  and  in  a  meaningful  manner. 

The  mean  KD  Range  training  effectiveness  rating  was  4.4,  which  was  significantly  greater  than  ISMT's 
mean  of  2.9.  Although  it  scored  lower  than  the  KD  Range,  ISMT's  score  indicates  that  it  is  still  a 
moderately  effective  trainer,  at  least.  Response  patterns  indicated  that  some  instructors  do  not  use 
ISMT  to  train  certain  categories  of  marksmanship,  such  as  ballistics  and  zeroing  tasks. 

Based  on  the  survey  results,  the  following  conclusions  and  recommendations  are  offered:  1)  The  survey 
exhibited  favorable  characteristics  as  a  marksmanship  trainer  evaluation  tool,  but  an  important  future 
step  would  be  to  compare  the  subjective  effectiveness  ratings  from  this  survey  to  objective  training 
results,  as  they  become  available.  2)  The  survey  identified  a  clear  SME  preference  for  the  KD  Range  over 
ISMT  as  a  training  tool  for  marksmanship  qualification.  3)  Despite  this  preference,  the  SMEs  still  viewed 
ISMT  as  a  useful  training  tool.  4)  Based  on  the  response  patterns  and  associations  between  training 
effectiveness,  task  importance,  and  task  difficulty,  the  KD  Range  should  be  used  to  train  items  that  are 
more  important  and  difficult,  such  as  aiming  tasks.  When  KD  Range  time  is  particularly  scarce,  a  good 
use  of  limited  assets  would  be  to  divert  less  important  and  less  difficult  marksmanship  tasks,  such  as 
Weapons  Handling  items,  to  ISMT.  5)  There  was  some  disagreement  among  SMEs  regarding  whether  or 
not  ISMT  was  used  to  train  certain  tasks  (e.g.,  Ballistics  and  Zeroing  tasks).  It  would  probably  be  useful 
for  USMC  training  experts  and  decision-makers  to  specifically  look  into  which  ISMT  features  are  used 
and  how  they  are  used,  and  conversely,  which  features  are  not  used,  and  why  they  are  not  used. 

Perhaps  some  ISMT  features  are  underutilized,  and/or  simple  improvements  can  be  made  to  improve 
ISMT's  ability  to  train  certain  tasks,  ultimately  easing  the  demand  on  KD  Ranges. 
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INTRODUCTION 


Marksmanship  is  a  fundamental  skill  for  all  Marines  to  master.  Ongoing  military  operations  require 
the  United  States  Marine  Corps  (USMC)  to  train  Marines  as  rapidly  and  effectively  as  possible,  but  the 
opportunity  to  train  on  live  fire  ranges  is  limited.  Marksmanship  simulators  such  as  the  Indoor 
Simulated  Marksmanship  Trainer  (ISMT)  may  help  relieve  some  of  this  pressure  on  traditional  live  fire 
ranges,  (a.k.a.,  Known  Distance  (KD)  ranges).  Simulators  can  offer  several  advantages  over  the 
traditional  KD  range.  Some  simulators  are  compact  and  portable  enough  to  allow  Marines  to  train  in 
settings  such  as  ships  or  embassies  that  are  too  confined  for  a  KD  Range  (Training  and  Education 
Command,  2010).  Simulators  are  typically  less  expensive  to  operate  because  they  save  on  the  cost  of 
ammunition,  targets,  and  other  supplies  (Training  and  Education  Command,  2010).  The  ability  to  place 
sensors  on  the  trainee's  weapon  also  allows  instructors  to  provide  more  specific  feedback  compared  to 
a  KD  Range,  where  the  instructor  may  not  be  able  to  pinpoint  deficiencies  in  technique  so  quickly  and 
easily. 

Despite  these  potential  benefits,  determining  whether  simulators  offer  training  that  is  comparable  or 
even  superior  to  a  KD  Range  can  be  difficult.  A  full  training  effectiveness  evaluation  (TEE)  comparing  a 
new  system  to  an  established  training  method  often  requires  a  substantial  amount  of  time  and 
resources,  and  can  disrupt  training  schedules.  Current  demands  on  the  training  pipeline  make  such 
evaluations  even  more  challenging.  The  amount  of  time  required  to  evaluate  a  new  system  can 
approach  the  time  required  to  design  and  build  it,  meaning  that  new  systems  come  online  before  the 
old  system  has  been  evaluated.  As  a  result,  improvement  can  be  sluggish  and  trainers  are  implemented 
without  a  full  understanding  of  the  new  system's  effectiveness  or  advantages  over  the  system  being 
replaced.  The  USMC  requires  a  method  to  evaluate  training  systems  that  is  faster,  less  expensive,  and 
less  disruptive  while  also  being  applicable  to  a  wide  range  of  simulators. 

The  goal  of  this  effort  is  to  determine  if  task  analytic  and  survey  techniques  can  be  used  to  evaluate 
the  effectiveness  of  training  systems  when  time  and  resources  do  not  permit  a  full-fledged  TEE.  This 
report  describes  the  development,  administration,  and  results  of  a  survey  comparing  the  KD  Range  and 
ISMT  as  marksmanship  training  systems.  The  results  of  the  survey,  which  was  administered  to 
marksmanship  subject  matter  experts  (SMEs),  will  eventually  be  compared  to  objective  training 
outcomes  from  a  companion  study,  as  those  results  become  available.  That  comparison  will  ultimately 
determine  the  utility  of  this  type  of  survey-based  evaluation  for  assessing  a  training  system's 
effectiveness. 


METHOD 

The  survey  developed  in  this  project  is  called  the  Marksmanship  Training  Survey  (MTS).  The  MTS  is 
based  upon  an  analysis  of  the  Marine  Corps  Rifle  Marksmanship  Manual  (USMC,  2001)  focusing  on  both 
the  KD  Range  and  ISMT  as  facilities  for  training  marksmanship.  Each  facility  is  described  below. 
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Known  Distance  Range 


The  KD  Range  is  a  rifle  range  with  targets  at  predetermined,  fixed  distances  (see  Figures  la  and  lb). 
Trainees  receive  marksmanship  instruction  in  training  phases  known  as  "tables".  Training  progresses 
through  the  following  tables: 

Fundamental  Rifle  Marksmanship  (Training  Table  1/1A)  -  This  table  trains  Marines  in  the 
fundamental  knowledge,  skills,  and  attitudes  necessary  to  fire  the  rifle  safely  and  accurately. 
Table  1  serves  as  the  foundation  upon  which  more  advanced  training  is  based.  Marines  learn  to 
shoot  from  various  firing  positions  (prone,  kneeling,  and  standing)  and  to  engage  targets  at  a 
sustained  rate  of  fire. 

Basic  Combat  Rifle  Marksmanship  (Training  Table  2)  -  This  table  begins  the  transition  from 
fundamental  marksmanship  skills  to  combat  marksmanship.  Marines  learn  to  execute  different 
weapon  carries  and  reloads,  and  to  engage  targets  using  controlled  pairs  of  shots.  Marines  also 
begin  to  engage  moving  targets. 

More  advanced  marksmanship  skills  such  as  using  combat  optics,  engaging  targets  while  the  Marine 
is  moving,  shooting  at  night,  and  engaging  targets  at  unknown  distances  are  trained  during  Intermediate 
Combat  Rifle  Marksmanship  (Training  Table  3)  and  Advanced  Combat  Rifle  Marksmanship  (Training 
Table  4).  These  training  tables  are  not  conducted  on  a  KD  Range,  however,  so  we  did  not  include  them 
in  our  evaluation. 

On  the  KD  Range,  trainees  fire  live  rounds  at  targets  in  order  to  practice  the  skills  necessary  for  Table 
1  and  Table  2  qualification.  Target  distances  for  the  M-16  include  25, 100,  200,  300,  and  500  yards, 
depending  on  the  training  table  being  conducted.  One  instructor  is  assigned  for  every  four  trainees  on 
the  KD  Range  (W.  Becker,  personal  communication,  August  23,  2012). 


Figure  la  and  lb.  Images  of  Known  Distance  Ranges. 
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ISMT 


ISMT  is  a  weapons  training  simulator  built  for  indoor  use  (see  Figure  2).  The  training  system  consists 
of  modified  M-16  rifles,  a  firing  line,  an  instructor  station,  and  an  audio/visual  system  for  displaying 
simulated  targets.  While  ISMT  is  compatible  with  a  wide  range  of  weapon  systems  for  training 
marksmanship  skill,  shoot/no-shoot  judgment,  and  weapons  tactics  (Training  and  Education  Command, 
2010;  Indoor  Simulated  Marksmanship  Trainer  -  Enhanced,  2008),  the  current  effort  focuses  on  ISMT  as 
used  for  Training  Tables  1  and  2  with  the  M-16  rifle.  For  training  purposes,  the  rifles  are  modified  to  fire 
laser  light  instead  of  live  ammunition.  Trainees  occupy  positions  along  the  firing  line,  and  a  large  display 
screen  is  located  20  feet  "downrange",  upon  which  the  simulated  targets  are  projected.  When  the 
trainee  fires  at  the  target,  ISMT  registers  and  records  the  point  of  impact  of  the  simulated  round  with  an 
accuracy  of  two  minutes  of  angle  (Yates,  2004).  Point  of  impact  and  important  marksmanship  variables 
that  can  affect  it,  such  as  point  of  aim,  trigger  pressure,  buttstock  pressure,  and  barrel  cant,  can  be 
displayed  immediately  to  the  instructor  and  student.  Trainee  scores  can  be  tracked  over  time  and  ISMT 
can  replay  a  trace  of  the  shooter's  point  of  aim  prior  to  and  immediately  after  firing  the  weapon, 
providing  feedback  to  the  trainee.  This  type  of  immediate  feedback  is  designed  to  help  instructors 
correct  trainees'  errors  and  improve  their  technique.  Training  in  ISMT  is  typically  conducted  with  one 
instructor  per  group  of  approximately  eight  trainees,  but  the  size  of  a  group  varies  based  on  the  size  of 
the  ISMT  facility  (W.  Becker,  personal  communication,  August  23,  2012). 


Figure  2.  The  ISMT  training  system. 

ISMT  is  able  to  reproduce  the  KD  Range  training  necessary  for  qualification  in  Training  Tables  1  and  2, 
with  the  exception  of  zeroing  the  weapon.  Trainees  cannot  zero  in  ISMT  because  the  system's  software 
adjusts  for  any  bias  in  the  weapon  automatically,  and  because  the  distance  between  the  display  screen 
and  the  shooter  is  too  small  to  allow  for  proper  training  in  zeroing  the  weapon  (W.  Becker,  personal 
communication,  August  23,  2012). 
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Marksmanship  Training  Survey  (MTS) 

We  developed  the  MTS  as  a  tool  for  SMEs  to  rate  both  ISMT  and  KD  Range  facilities  on  their  abilities 
to  train  marksmanship  tasks.  The  MTS  can  be  found  in  Appendix  A.  The  survey  was  developed  by 
analyzing  the  Marine  Corps  Rifle  Marksmanship  Manual  and  dividing  the  marksmanship  domain  into 
tasks,  secondary  tasks,  and  tertiary  tasks.  Chapter  titles  from  the  Marksmanship  Manual  served  as  tasks 
(e.g.,  Introduction  to  Rifle;  Weapons  Handling;  Fundamentals  of  Marksmanship).  These  tasks  were  the 
highest  level  in  the  task  hierarchy.  Chapter  sections  served  as  secondary  tasks  (e.g.,  trigger  control, 
breath  control,  aiming).  Topics  within  the  chapter  sections  were  identified  as  tertiary  tasks  (e.g.,  sight 
alignment,  sight  picture).  Tertiary  tasks  were  identified  as  the  actual  actions  carried  out  by  Marines 
firing  a  weapon.  An  example  of  a  full  hierarchical  classification  from  task,  to  secondary  task,  to  tertiary 
task  would  be: 

Fundamentals  of  marksmanship  -  Aiming  -  Sight  alignment 

The  Marine  Corps  Rifle  Marksmanship  Manual  contains  fairly  detailed  descriptions  of  tertiary  tasks. 
These  lengthy  descriptions  were  converted  to  simple  action  statements  in  order  to  make  them  more 
suitable  for  inclusion  in  the  survey.  With  sight  alignment,  for  example,  we  translated  "Sight  alignment  is 
the  relationship  between  the  front  sight  post  and  rear  sight  aperture  and  the  aiming  eye"  into  a  survey 
item  that  read  "Establish  the  correct  sight  alignment".  Survey  participants  could,  however,  recover  the 
detailed  descriptions  of  each  item  if  they  wanted  more  information,  as  follows.  The  MTS  was 
constructed  as  a  worksheet  in  an  Excel  workbook;  by  hovering  the  mouse  cursor  over  an  item,  its 
detailed  description  would  appear  in  a  pop-up  box. 

In  addition  to  the  survey  worksheet,  the  workbook  contained  a  worksheet  for  Instructions,  Informed 
Consent,  Privacy  Act,  and  Marksmanship  Experience.  The  Marksmanship  Experience  worksheet  (refer  to 
Appendix  A)  contained  questions  about  which  weapons  the  SMEs  trained  others  to  use,  amount  of 
experience  as  an  M-16  rifleman  and  as  an  M-16  instructor,  and  amount  of  ISMT  experience.  It  also 
provided  space  for  general  open-ended  comments. 

Long  surveys  tend  to  have  low  response  rates  and  suffer  from  poor  response  quality  (Galesic  & 
Bosnjak,  2009).  In  order  to  manage  the  length  of  the  MTS,  we  excluded  items  from  the  Marine  Corps 
Rifle  Marksmanship  Manual  that  were  not  directly  related  to  the  shooter  accurately  engaging  a  target. 
For  example,  we  excluded  items  about  learning  the  specifics  of  how  the  rifle's  gas  system  operates,  rifle 
maintenance,  loading  and  storing  magazines,  or  rifle  carry  and  transport  positions.  We  enlisted  the  help 
of  a  local  SME,  who  has  qualified  in  marksmanship  at  the  expert  level,  to  eliminate  these  types  of  items 
and  to  ensure  the  use  of  proper  terminology.  The  final  version  of  the  MTS  (see  Appendix  A)  contained 
48  tertiary  task  statements,  plus  two  questions  about  the  overall  effectiveness  of  ISMT  and  KD  Range  for 
training  marksmanship. 

Each  of  the  48  tertiary  survey  items  was  rated  on  the  following  four  dimensions: 
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1)  Importance  for  Accuracy,  defined  in  the  survey  instructions  as  "The  degree  to  which  incorrect 
performance  of  the  task  would  result  in  reduced  ability  to  place  rounds  effectively".  Importance  was 
rated  from  1  (not  important)  to  5  (extremely  important),  or  not  applicable  (NA). 

2)  Difficulty  to  Learn,  defined  as:  "Difficulty  to  learn  reflects  the  total  amount  of  time  and  effort 
required  to  learn  to  perform  a  task  successfully  and  independently,  relative  to  all  other  marksmanship 
training".  Difficulty  was  rated  from  1  (one  of  the  easiest  tasks  to  learn)  to  5  (one  of  the  most  difficult  to 
learn  of  all  tasks),  or  NA. 

3)  ISMT Effectiveness,  defined  by  the  question:  "How  effective  is  ISMT  in  training  the  Marine  to  perform 
this  task?".  ISMT  effectiveness  was  rated  from  1  (not  effective)  to  5  (extremely  effective),  or  NA  (the 
task  is  not  trained  using  this  method). 

4)  KD  Range  Effectiveness,  defined  by  the  question:  "How  effective  is  the  KD  Range  in  training  the 
Marine  to  perform  this  task?".  KD  Range  effectiveness  was  rated  from  1  (not  effective)  to  5  (extremely 
effective),  or  NA  (the  task  is  not  trained  using  this  method). 

Each  of  the  48  items  also  provided  space  for  general  open-ended  comments. 

The  two  final  questions  on  the  survey  asked  participants  to  rate  the  overall  effectiveness  of  ISMT,  as 
well  as  the  overall  effectiveness  of  the  KD  Range,  for  training  marksmanship.  The  effectiveness  of  each 
method  was  rated  from  1  (not  effective)  to  5  (extremely  effective).  The  entire  set  of  instructions  with  all 
of  the  response  options  can  be  found  in  the  MTS  in  Appendix  A. 

Survey  Administration  Procedure 

With  prior  agreement  and  arrangement,  an  electronic  copy  of  the  survey  was  e-mailed  to  a  point  of 
contact  (POC)  at  the  Weapons  Training  Battalion  Quantico,  Virginia.  The  POC  then  forwarded  the  survey 
to  39  USMC  M-16  marksmanship  SMEs  (instructors).  After  reading  through  the  "Instructions", 

"Informed  Consent",  and  "Privacy  Act"  worksheets,  the  SMEs  continued  on  to  fill  out  the 
"Marksmanship  Experience"  and  "Survey"  worksheets.  The  SMEs  then  sent  completed  surveys  back  to 
the  POC,  who  in  turn  forwarded  them  to  the  research  team  at  NAMRU-D. 

Participants 

Twenty-six  of  the  39  marksmanship  instructors  completed  and  returned  the  MTS,  for  a  response  rate 
of  67%.  Three  surveys  were  dropped  due  to  a  probable  file-saving-error  that  resulted  in  identical 
responses  on  each  of  194  answers.  One  additional  survey  was  dropped  because  the  SME  reported  no 
experience  with  ISMT.  The  remaining  22  SMEs  had  a  mean  of  6.0  years  of  experience  with  the  M-16 
rifle,  and  had  been  training  others  to  use  the  M-16  for  an  average  of  2.6  years.  SMEs  had  an  average  of 
2.2  years  of  experience  working  with  ISMT. 
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ANALYSES  AND  RESULTS 


In  order  to  facilitate  interpretation  of  this  large  dataset,  we  grouped  the  48  tertiary  items  into  the 
following  six  marksmanship  categories:  Weapons  Handling,  Firing  Positions,  Aiming,  Trigger  Control, 
Ballistics,  and  Zeroing.  These  groups  were  guided  by,  and  corresponded  well  to,  chapter  sections  within 
the  Marine  Corps  Rifle  Marksmanship  Manual.  These  six  categories  and  the  tertiary  tasks  that  they 
contain  are  listed  in  Appendix  B. 


Interrater  Agreement 

The  MTS  data  were  first  analyzed  to  gauge  interrater  agreement,  since  the  usefulness  of  a  subjective 
assessment  tool  like  this  one  is  limited  by  the  extent  to  which  expert  raters  can  agree  upon  task  ratings. 
A  high  degree  of  disagreement  would  indicate  that  the  outcome  of  the  assessment  will  be  dependent  on 
the  particular  sample  of  raters  selected,  limiting  the  utility  of  the  evaluation  tool.  Assessing  agreement 
can  be  facilitated  by  the  use  of  multiple  methods,  as  different  indices  tend  to  yield  convergent  results 
(LeBreton  &  Senter,  2008).  Although  the  choice  of  index  is  largely  a  matter  of  personal  preference 
(LeBreton  &  Senter,  2008),  we  used  the  rwg  and  Average  Deviation  (AD)  indices  of  agreement  per  item 
following  the  advice  of  Burke,  Finkelstein,  and  Dusig  (1999).  The  rwg  statistic  examines  the  distribution  of 
raters'  responses  compared  to  a  hypothetical  null  distribution.  We  used  a  null  distribution  composed  of 
uniform  random  responses  because  we  lacked  a  sound  basis  to  identify  subgroups  of  raters  a  priori 
(LeBreton  &  Senter,  2008)  or  determine  the  nature  of  any  potential  ratings  bias  and  could  not  otherwise 
model  the  response  variability  (Burke,  Finkelstein,  &  Dusig,  1999).  Interrater  agreement  was  tested  at 
the  p  =  .05  level. 

The  AD  statistic  evaluates  the  average  deviation  of  each  rater  from  the  mean  or  median  of  all  raters. 
We  computed  the  AD  statistic  as  an  additional  measure  of  agreement  to  strengthen  our  interpretation 
of  the  rwg  statistic.  Significance  of  agreement  was  tested  at  the  p  =  .05  level  for  individual  items  using 
established  critical  values  (Dunlap,  Burke,  &  Smith-Crowe,  2003). 

The  AD  statistic  can  be  computed  based  on  either  mean  rating  scores  or  median  rating  scores.  For 
the  purposes  of  measuring  agreement,  we  computed  the  AD  statistic  using  median  rating  scores 
because  these  values  can  be  more  robust  and  sensitive  in  detecting  agreement  than  values  computed 
using  the  mean  (Burke  et  al.,  1999).  Appendix  C  presents  both  the  rwg  and  AD  values  for  each  item,  as 
well  as  the  critical  values  to  reach  significance  at  the  p  =  .05  level. 

To  evaluate  aggregated  marksmanship  categories,  we  examined  the  number  of  individual  items 
showing  agreement  out  of  the  total  number  of  items  in  that  category.  There  is  no  widely  accepted 
significance  test  for  the  rwg  and  AD  statistics  at  the  aggregated  level. 

In  our  MTS  dataset  we  found  that  SMEs  agreed  in  their  task  ratings  as  a  whole,  but  agreement  varied 
between  marksmanship  categories  and  on  different  dimensions  (Importance,  Difficulty,  ISMT 
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Effectiveness,  and  KD  Range  Effectiveness).  These  variations  can  affect  the  interpretability  of  the  findings 
and  will  be  discussed  in  the  appropriate  section  of  the  results. 

Overall  Results 

To  gain  a  composite  picture  of  the  Importance  and  Difficulty  ratings  for  the  48  tertiary  tasks,  we 
calculated  the  overall  mean  for  each  of  these  two  ratings.  Across  the  48  items,  the  22  SMEs  gave 
Importance  a  mean  rating  of  4.31  (SD  =  0.33).  The  mean  rating  for  Difficulty  was  2.84  (SD  =  0.58).  The 
"Top  Five"  and  "Bottom  Five"  items  for  Importance  and  Difficulty  are  rank  ordered  in  Tables  1  and  2, 
respectively.  The  entire  rank  orderings  for  Importance  and  Difficulty  are  presented  in  Appendices  D  and 
E,  respectively. 


Table  1 

Items  Rated  Most  Important  for  Accuracy  and  Least  Important  for  Accuracy 


Importance 

Rank 

Item  # 

Task  statement 

Category 

Mean 

1 

35 

Establish  battlesight  zero 

Zeroing 

4.91 

2 

12 

Apply  marksmanship  fundamentals  until  the 
round  exits  the  barrel  of  the  rifle 

Aiming 

4.86 

3 

1 

Establish  the  correct  sight  alignment 

Aiming 

4.82 

4 

2 

Establish  the  correct  sight  picture 

Aiming 

4.82 

5 

5 

Maintain  sight  alignment  and  sight  picture 

Aiming 

4.73 

44 

3 

Establish  proper  stock  weld 

Aiming 

3.91 

45 

15 

Don  the  loop  sling 

Weapons 

handling 

3.68 

46 

19 

Use  proper  positioning  of  the  right  elbow 

Weapons 

handling 

3.68 

47 

41 

Use  front  sight  post  method  to  determine 
distance  to  the  target 

Ballistics 

3.59 

48 

11 

Maintain  proper  interrupted  trigger  control 

Trigger  control 

3.55 

10 


Table  2 


Items  Rated  Most  Difficult  to  Learn  and  Least  Difficult  to  Learn 


Difficulty 

Rank 

Item  # 

Task  statement 

Category 

Mean 

1 

44 

Use  multiple  methods  to  determine  distance 
to  the  target 

Ballistics 

4.14 

2 

48 

Engage  targets  while  wearing  the  field 
protective  mask 

Aiming 

4.06 

3 

10 

Maintain  proper  uninterrupted  trigger 
control 

Trigger  control 

4.05 

Shift  point  of  aim  or  use  offset  aiming  when 

4 

37 

conditions  do  not  permit  mechanical  sight 
adjustment 

Aiming 

3.86 

5 

40 

Use  unit  of  measure  method  to  determine 
distance  to  the  target 

Ballistics 

3.86 

44 

15 

Don  the  loop  sling 

Weapons 

handling 

2.27 

45 

19 

Use  proper  positioning  of  the  right  elbow 

Weapons 

handling 

2.18 

46 

25 

Assume  the  sitting  position 

Firing  positions 

2.18 

47 

3 

Establish  proper  stock  weld 

Aiming 

2.00 

48 

24 

Assume  the  prone  position 

Firing  positions 

1.82 

The  two  overarching  questions  at  the  end  of  the  survey  asked  SMEs  to  provide  an  overall  rating  of  KD 
Range  Effectiveness,  as  well  as  ISMT  Effectiveness.  The  two  overall  questions  demonstrated  good  rater 
agreement  (rwg  and  AD  were  0.61  and  0.82  for  ISMT  Effectiveness,  and  0.92  and  0.41  for  KD  Range 
Effectiveness,  respectively).  The  mean  rating  for  the  KD  Range  Effectiveness  was  4.81  (SD  =  0.40),  versus 
3.52  (SD  =  0.87)  for  ISMT  Effectiveness.  A  paired  sample  f-test  showed  that  this  difference  was 
significant,  f(20)  =  6.16,  p  <  0.01. 

A  second  way  to  compare  KD  Range  and  ISMT  Effectiveness  ratings  is  to  look  at  means  for  these 
variables  across  all  22  respondents  and  all  48  tertiary  tasks.  When  calculated  in  this  manner,  the  KD 
Range  still  had  the  higher  rating,  but  ratings  for  each  facility  were  lower.  The  mean  effectiveness  rating 
for  the  KD  Range  dropped  somewhat  to  4.40  (SD  =  0.24),  and  that  for  ISMT  dropped  to  2.92,  (SD  =  0.61). 
The  overall  mean  ISMT  Effectiveness  ratings  must  be  interpreted  with  caution,  however,  since  they 
demonstrated  poor  rater  agreement  in  the  Firing  Position  and  Ballistics  task  categories  (discussed 
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below).  In  a  later  section  we  also  apply  f-tests  to  these  KD  Range  vs.  ISMT  differences  within  each  of  the 
six  marksmanship  categories. 

Analysis  by  Marksmanship  Category 

For  each  of  the  six  marksmanship  categories,  we  calculated  the  average  rating  of  Importance  and 
Difficulty,  and  these  data  are  graphed  in  Figures  3  and  4,  respectively.  Similarly  for  each  category,  mean 
ratings  for  KD  Range  Effectiveness  versus  ISMT  Effectiveness  are  shown  in  Figure  5.  In  all  cases,  means 
were  calculated  across  all  subjects  and  items  within  each  category. 
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4.52 


Trigger 

control 

Ballistics 

Zeroing 


4.11 

4.08 


4.58 


0  1  2  3  4  5 


Importance 


Figure  3.  Average  Ratings  of  Importance 
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Figure  4.  Average  Ratings  of  Difficulty 
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Figure  5.  Average  Ratings  of  KD  Range  Effectiveness  and  ISMT  effectiveness 

Each  of  the  48  tertiary  survey  items  had  “Not  Applicable:  The  task  is  not  trained  using  this  method"  as 
a  valid  response  choice.  Respondents  invoked  this  choice  to  varying  degrees,  and  as  a  result,  certain 
questions  did  not  receive  a  numeric  rating  from  some  of  the  respondents.  NA  ratings  were  treated  as 
strictly  qualitative  data  and  therefore  did  not  enter  into  the  calculation  of  means,  f-tests,  etc.  Figures  6 
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and  7  show  the  percentage  of  questions  within  each  category  that  received  NA  responses  for  the 
Importance  and  Difficulty  ratings,  respectively.  Likewise,  Figure  8  shows  these  percentages  for  KD  Range 
Effectiveness  and  ISMT Effectiveness  ratings.  NA  responses  were  mostly  reserved  for  ISMT Effectiveness 
ratings  for  the  Zeroing  and  Ballistics  categories,  indicating  that  in  the  experience  of  some  SMEs,  ISMT  is 
not  used  to  train  these  components  of  marksmanship. 
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Figure  6.  Percentage  of  NA  Ratings  for  Importance 
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Figure  7.  Percentage  of  NA  Ratings  for  Difficulty 
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Figure  8.  Percentage  of  NA  Ratings  for  KD  Range  Effectiveness  and  ISMT  Effectiveness 

The  NA  ratings  are  informative  since  they  are  indicators  of  which  items  are  not  currently  trained  by  a 
particular  method,  but  they  are  also  problematic  in  that  they  effectively  become  missing  data  for 
comparing  training  methods.  This  is  especially  true  here  for  paired  sample  f-tests  comparing  ratings  of 
ISMT  vs.  KD  Range  effectiveness.  For  example,  there  were  five  survey  items  on  Ballistics,  and  each  of 
the  22  participants  provided  a  rating  of  KD  Range  training  effectiveness  for  each  item,  for  a  total  of  110 
responses.  None  of  those  responses  were  “NA".  All  participants  also  rated  each  item  for  ISMT 
effectiveness,  but  35  of  those  responses  were  "NA",  meaning  that  for  35  of  the  response  pairs  ( ISMT  vs. 
KD  Range  Effectiveness)  we  could  not  make  a  meaningful  comparison.  That  is,  a  paired  sample  f-test 
could  not  calculate  a  difference  score  for  those  particular  pairs.  However,  75  "valid"  pairs  still  remained. 
To  conduct  the  paired  sample  f-test,  we  chose  a  conservative  approach  of  omitting  a  participant's 
response  pair  if  it  contained  an  "NA",  versus  another  approach  such  as  using  that  participant's  mean 
(perhaps  calculated  from  only  a  small  number  of  responses)  of  non-NA  responses  within  that  category. 
This  approach  still  allowed  for  comparison  of  75  response  pairs  within  the  Ballistics  category,  and  this 
was  the  "worst  case"  category  in  terms  of  having  the  highest  percentage  (31.8%)  of  omitted  pairs 
(Figure  8). 

The  KD  Range  Effectiveness  and  ISMT  Effectiveness  ratings  presented  in  Figure  5  were  recalculated 
for  the  omitted  pairs  data  set,  and  these  new  means  are  presented  in  Figure  9.  Comparison  of  these 
two  figures  shows  that  there  was  little  or  no  effect  of  omitting  pairs  except  for  the  Zeroing  and  Ballistics 
categories,  and  then  only  for  the  KD  Range  data,  where  the  means  increased  slightly.  ISMT  data  for 
these  two  categories  showed  little  or  no  change  since  the  vast  majority  of  NA  ratings  were  due  to  ISMT, 
and  since  NA  ratings  were  considered  qualitative  data  they  did  not  factor  into  means  to  begin  with. 
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Figure  9.  Average  Ratings  of  KD  Range  Effectiveness  and  ISMT  effectiveness  with  Missing  Data  Pairs 
Omitted 

Mean  Importance,  Difficulty,  ISMT  Effectiveness,  and  KD  Range  Effectiveness  means  are  reviewed 
below  for  each  of  the  six  marksmanship  categories.  Paired  sample  t- tests  results  are  also  presented  to 
compare  effectiveness  ratings  of  KD  Range  versus  ISMT  training.  Because  each  of  the  six  marksmanship 
categories  was  composed  of  a  different  number  of  survey  items,  and  because  the  number  of  omitted 
pairs  varied,  degrees  of  freedom  for  each  t  value  vary  as  well.  Ratings  within  each  category 
demonstrated  good  agreement  unless  otherwise  noted. 

Weapons  Flandling 

Weapons  Handling  received  a  mean  Importance  rating  of  4.08  (SD  =  0.86)  and  a  mean  Difficulty  rating 
of  2.67  (SD  =  0.98).  Mean  Importance  and  Difficulty  ratings  are  shown  in  Figures  3  and  4,  respectively. 
For  the  KD  Range  versus  ISMT  effectiveness  comparison,  a  paired  samples  f-test  demonstrated  that  the 
KD  Range  (M  =  4.22,  SD  =  0.75)  was  rated  as  more  effective  at  training  Weapons  Handling  tasks  than 
ISMT  (M  =  2.97,  SD  =  1.00),  with  f(188)  =  16.12,  p  <  .001  (see  Figure  9). 

Firing  Positions 

Firing  Positions  received  a  mean  Importance  rating  of  4.34  (SD  =  0.89)  and  a  mean  Difficulty  rating  of 
2.60  (SD  =  1.23).  Only  four  out  of  eight  items  demonstrated  good  agreement  for  Difficulty  ratings  on 
both  rwg  and  AD,  with  an  additional  item  showing  good  agreement  on  the  AD  index  but  not  rwg.  Difficulty 
ratings  for  Firing  Positions  as  a  category  therefore  demonstrate  only  modest  agreement  and  should  be 
interpreted  carefully.  Regarding  effectiveness  ratings,  a  paired  samples  f-test  demonstrated  that  the  KD 
Range  (M  =  4.49,  SD  =  0.73)  was  rated  as  more  effective  at  training  Firing  Position  tasks  than  ISMT  (M  = 
3.16,  SD  =  1.24),  with  f(165)  =  11.78,  p  <  .001  (see  Figure  9).  These  results  should  be  interpreted  with 
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caution,  however,  since  none  of  the  eight  items  in  the  Firing  Positions  category  demonstrated 
agreement  among  the  SMEs  on  ISMT  Effectiveness  ratings. 

Aiming 

Aiming  received  a  mean  Importance  rating  of  4.52  (SD  =  0.85)  and  a  mean  Difficulty  rating  of  3.17  (SD 
=  1.12).  For  the  Effectiveness  comparison,  a  paired  samples  f-test  demonstrated  that  the  KD  Range  (M  = 
4.56,  SD  =  0.67)  was  rated  as  more  effective  at  training  Aiming  tasks  than  ISMT  (M  =  3.28,  SD  =  1.15), 
with  f(246)  =  16.23,  p  <  .001  (see  Figure  9). 

Trigger  Control 

Trigger  Control  received  a  mean  Importance  rating  of  4.11  (SD  =  0.97)  and  a  mean  Difficulty  rating  of 
2.96,  (SD  =  1.08).  A  paired  samples  f-test  demonstrated  that  the  KD  Range  ( M  =  4.34,  SD  =  0.77)  was 
rated  as  more  effective  at  training  Trigger  Control  tasks  than  ISMT  ( M  =  3.48,  SD  =  1.02),  with  f(131)  = 
9.44,  p  <  .001  (see  Figure  9). 

Ballistics 

Ballistics  received  a  mean  Importance  rating  of  4.08,  SD  =  0.94  and  a  mean  Difficulty  rating  of  3.84,  SD 
=  0.90.  A  paired  samples  f-test  demonstrated  that  the  KD  Range  (M  =  4.32,  SD  =  0.84)  was  rated  as 
more  effective  at  training  Ballistics  tasks  than  ISMT  (M  =  2.16,  SD  =  1.23),  with  f(74)  =  12.80,  p  <  .001 
(see  Figure  9).  ISMT  Effectiveness  ratings  should  be  interpreted  with  caution,  as  only  one  out  of  five 
Ballistics  items  demonstrated  good  agreement. 

Zeroing 

Zeroing  received  a  mean  Importance  rating  of  4.58,  SD  =  0.76  and  a  mean  Difficulty  rating  of  2.98,  SD 
=  1.13.  A  paired  samples  t-test  demonstrated  that  the  KD  Range  (M  =  4.66,  SD  =  0.61)  was  rated  as 
more  effective  at  training  Zeroing  tasks  than  ISMT  ( M  =  2.23,  SD  =  1.10),  with  f(124)  =  20.02,  p  <  .001 
(see  Figure  9). 

Correlations 

In  an  exploratory  effort  to  identify  any  associations  between  the  dimensions  of  item  Importance, 
Difficulty,  ISMT  Effectiveness,  and  KD  Range  Effectiveness,  we  constructed  the  correlation  matrix  for 
these  four  dimensions.  This  matrix  is  shown  in  Table  3.  The  significant  positive  correlation  between 
Importance  and  KD  Range  Effectiveness  was  fairly  strong  (r  =  .68,  p  <  .001);  items  rated  higher  in 
Importance  also  tended  to  be  rated  high  for  KD  Range  Effectiveness.  The  negative  correlation  between 
Difficulty  and  ISMT  Effectiveness  was  moderate  but  not  quite  significant  at  the  p  =  .05  level  (r  =  -.41,  p  = 
.06),  indicating  that  more  difficult  items  tended  to  receive  lower  ISMT  Effectiveness  ratings.  There  was 
some  indication  of  an  association  between  Importance  and  Difficulty  ratings,  but  it  was  not  statistically 
significant  (r  =  .37,  p  =  .10).  None  of  the  other  correlations  approached  significance. 
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Table  3 

Correlations  between  Importance,  Difficulty,  KD  Range  Effectiveness,  and  ISMT 
Effectiveness 

Importance  Difficulty  KD  Range  Effectiveness 
Difficulty  0.37 


KD  Range  Effectiveness  -0.20  -0.41 

ISMT  Effectiveness  0.68***  -0.01  -0.29 

***p  <  .001,  two-tailed 


Open-Ended  Comments 

Five  SMEs  provided  a  total  of  11  general  open-ended  comments.  The  comments  were  wide-ranging  and 
included  suggestions  for  useful  teaching  aids,  general  areas  of  difficulty  for  students  (e.g.,  sight 
alignment,  eye  relief),  and  areas  where  ISMT  might  be  improved  (e.g.,  more  realistic  recoil,  ability  to 
teach  various  shooting  positions).  All  of  the  comments  are  presented  in  Appendix  F. 

DISCUSSION 

The  goal  of  this  effort  was  to  determine  the  utility  of  a  quickly  administered  subjective  evaluation 
tool  for  assessing  the  effectiveness  of  marksmanship  trainers  and  as  a  possible  alternative  to 
comprehensive  TEE's.  Using  a  task  analytic  approach,  marksmanship  training  materials,  and  SME  input, 
we  developed  a  survey  which  instructors  and  SMEs  could  complete  in  under  30  minutes.  Twenty-two 
USMC  marksmanship  instructors  returned  valid  surveys  and  we  discuss  those  results  here.  These  data 
will  eventually  be  analyzed  further  to  see  if  they  are  predictive  of  the  objective  training  results  of  a 
companion  study  comparing  the  effectiveness  of  ISMT  and  KD  Range  training  methods,  when  those 
results  become  available. 

Interrater  Agreement 

Overall  interrater  agreement  was  good,  indicating  reliable  results,  with  a  few  exceptions.  Agreement 
was  low  for  ISMT  Effectiveness  ratings  within  the  Firing  Positions  and  Ballistics  categories,  which  makes 
those  results  more  difficult  to  interpret.  Although  low  agreement  makes  system  evaluation  more 
challenging,  it  does  not  necessarily  mean  that  an  item  is  not  informative.  Low  agreement  could  indicate 
legitimately  different  groups  of  raters,  as  would  result  from  different  instructors  using  the  system 
differently,  for  example.  Perhaps  some  SMEs  used  ISMT  to  train  some  tasks  while  other  SMEs  did  not. 
SMEs  who  did  not  use  ISMT  to  train  certain  tasks  may  have  rated  it  poorly  or  NA,  while  SMEs  who  used 
ISMT  rated  it  highly.  This  explanation  seems  plausible  given  the  high  agreement  among  SMEs  on  KD 
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Range  effectiveness,  but  low  agreement  among  the  same  SMEs  in  certain  categories  on  ISMT 
effectiveness.  There  may  be  some  inconsistencies  in  ISMT's  use,  leading  to  different  training  depending 
on  a  particular  instructor  or  facility.  Future  work  should  investigate  how  ISMT  is  actually  used  by 
different  instructors  in  order  to  identify  best  practices  to  improve  standardization  and  training 
effectiveness. 

Agreement  was  also  low  for  Difficulty  ratings  for  Firing  Positions,  but  it  is  not  clear  why.  One  possible 
explanation  is  again  that  instructors  are  using  the  systems  differently.  These  differences  could  have 
caused  some  trainees  to  learn  the  tasks  more  easily  than  others,  depending  on  how  instructors  used  the 
systems. 

Agreement  scores  must  also  be  interpreted  with  special  care  for  items  or  categories  with  a  high 
number  of  NA  responses,  as  was  the  case  with  ISMT  Effectiveness  ratings  for  the  Ballistics  and  Zeroing 
categories.  Numerous  NA  responses  in  the  presence  of  other  ratings  may  indicate  poor  agreement 
despite  good  values  for  rwg  and  AD.  Any  follow  on  work  should  try  to  determine  why  some  SMEs  feel 
that  certain  items  are  NA,  whereas  other  SMEs  have  no  problems  in  evaluating  those  same  items. 

Importance  Ratings 

Overall,  the  48  survey  items,  or  tertiary  tasks,  received  high  ratings  for  Importance.  The  mean 
Importance  rating  across  all  items  was  4.31,  and  the  mean  rating  for  each  of  six  categories  was  above  4 
(see  Figure  3),  which  corresponded  to  "FHighly  Important".  High  Importance  ratings  are  not  surprising 
here  since  all  items  on  the  survey  can  be  traced  back  to  the  Marine  Corps  Rifle  Marksmanship  Manual, 
which  focuses  on  tasks  important  to  marksmanship.  The  top  five  items  for  Importance  (see  Table  1) 
were  all  from  the  categories  of  Zeroing  or  Aiming.  This  result  makes  sense  since  a  rifle  that  is  not 
properly  zeroed  and  properly  aimed  cannot  be  expected  to  place  a  round  accurately  on  target 
(especially  the  first  round,  when  there  is  not  yet  any  feedback  on  point  of  impact).  The  item  with  the 
highest  Importance  rating,  "Establish  battlesight  zero",  had  a  mean  rating  of  4.91. 

The  bottom  five  items  for  Importance  in  Table  1  were  a  bit  more  heterogeneous,  originating  from  the 
categories  of  Weapons  Handling,  Trigger  Control,  Ballistics,  and  Aiming.  It  is  worth  noting  that  even  the 
item  with  the  lowest  rank  order  Importance  rating,  "Maintain  proper  interrupted  trigger  control",  still 
received  a  mean  rating  3.55.  This  falls  approximately  midway  between  the  descriptors  "3  -  Moderately 
Important"  and  "4  -  Highly  Important",  again  indicating  that  all  of  the  survey  items  were  viewed  as 
important  for  marksmanship. 

Difficulty  Ratings 

The  overall  mean  Difficulty  rating  was  2.84,  or  just  below  the  midpoint  descriptor  "3  -  Approximately 
half  the  tasks  are  easier  to  learn  and  half  are  more  difficult  to  learn".  The  item  rated  as  most  difficult, 
"Use  multiple  methods  to  determine  distance  to  the  target",  had  a  mean  rating  of  4.14.  A  rating  of  "4" 
corresponded  to  "Harder  to  learn  than  most  other  tasks".  The  item  with  the  lowest  Difficulty  rating  was 
"Assume  the  prone  position",  which  received  a  mean  rating  of  1.82,  or  close  to  a  rating  of  "2  -  Easier  to 
learn  than  most  other  tasks".  For  Difficulty  ratings,  SMEs  used  a  fairly  wide  range  of  the  available  scale, 
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and  their  average  rating  was  close  to  the  scale  midpoint.  These  two  characteristics  of  the  data  indicate 
that  with  the  exception  of  Firing  Position  Difficulty  ratings  as  discussed  above,  the  SMEs  were  able  to 
successfully  rate  tasks  relative  to  each  other  and  in  a  meaningful  manner. 

The  top  five  items  for  Difficulty  (see  Table  2)  were  from  the  categories  of  Ballistics,  Aiming,  and 
Trigger  Control.  Items  in  the  Ballistics  category  involved  making  distance  judgments,  which  is  known  to 
be  a  difficult  perceptual  task  (Allen  &  Rashotte,  2006).  The  Aiming  and  Trigger  Control  categories 
contained  challenging  items  that  require  fine  motor  control  and  precise  hand-eye  coordination,  so  it  is 
logical  that  these  items  and  categories  were  rated  among  the  most  difficult. 

The  bottom  five  items  for  Difficulty  in  Table  2  were  from  the  categories  of  Firing  Positions,  Aiming, 
and  Weapons  Handling.  These  five  items  involved  fairly  gross  motor  movement  and  positioning  tasks 
(e.g.,  "Assume  the  prone  position").  It  is  likely  that  the  gross  motor  and  fairly  simple  procedural  nature 
of  these  tasks  (especially  as  compared  to  the  five  most  difficult  items)  resulted  in  their  lower  Difficulty 
ratings. 

ISMT  vs.  KD  Range  Ratings 
Overall  Effectiveness  Ratings 

When  SMEs  were  asked  to  provide  one  overall  rating  for  KD  Range  Effectiveness  and  one  for  ISMT 
Effectiveness,  they  clearly  rated  the  KD  Range  (M  =  4.81)  as  more  effective  than  ISMT  (M  =  3.52).  The 
same  pattern  of  results  emerged  when  Effectiveness  means  were  calculated  across  all  48  tertiary  tasks. 
The  mean  for  the  KD  Range  was  4.40  while  that  for  ISMT  was  2.92.  It  is  unclear  exactly  why  ratings  were 
lower  when  means  were  calculated  across  the  48  individual  tertiary  items.  Perhaps  the  increased  detail 
and  specificity  at  the  item  level  prompted  SMEs  to  think  more  thoroughly  and  critically  in  their  ratings. 
What  is  clear  is  that  the  SMEs  consistently  rated  the  KD  Range  as  more  effective. 

There  are  at  least  two  potential  explanations  for  the  higher  KD  Range  Effectiveness  ratings.  First, 
since  Marines  are  tested  on  the  KD  Range  itself,  not  ISMT,  for  their  marksmanship  qualification,  the  KD 
Range  is  certainly  a  good  choice  for  learning  and  practicing  the  required  skills.  For  example,  on  the  KD 
Range  Marines  learn  to  deal  with  real  world  weather  effects  (e.g.,  wind,  atmospheric  obscuration,  etc.) 
and  experience  the  actual  noise  and  recoil  of  firing  live  ammunition  in  their  own  weapons.  While  the 
weapons  converted  for  ISMT  do  produce  some  recoil,  it  is  only  about  one-third  as  strong  as  that 
produced  by  live  ammunition,  and  one  SME  did  comment  on  the  non-realistic  recoil  provided  by  ISMT. 
Thus  it  may  be  argued  that  tasks  like  re-establishing  sight  alignment  and  sight  picture  are  not  as 
challenging  or  realistic  with  ISMT,  and  therefore  the  KD  Range  produces  better  training  for  qualification. 

A  second  factor  that  may  have  contributed  to  a  higher  rating  for  the  KD  Range  is  a  subjective 
preference  or  bias  for  firing  actual,  versus  simulated,  weapons  and  ammunition.  Simply  put,  live  fire  is 
probably  more  rewarding  and  fun  for  the  Marines  than  ISMT,  and  this  may  be  a  bias  that  affects  their 
ratings. 
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One  point  that  should  be  kept  in  mind  is  that  even  though  ISMT  received  lower  mean  Effectiveness 
ratings  than  the  KD  Range,  it  still  received  intermediate  to  favorable  ratings.  ISMT's  mean  rating  of  2.92 
across  the  48  items  is  slightly  below,  but  still  close  to,  a  rating  of"  3  -  Moderately  Effective".  ISMT's 
mean  score  of  3.52  for  the  overall  question  is  a  favorable  rating,  falling  between  "Moderately  Effective" 
and  "Highly  Effective".  The  SMEs  still  see  ISMT  as  a  useful  training  tool. 


Effectiveness  Ratings  by  Marksmanship  Categories 

In  addition  to  the  overall  KD  Range  Effectiveness  versus  ISMT  Effectiveness  ratings,  we  analyzed  the 
effectiveness  ratings  within  each  of  the  six  marksmanship  categories  (Figure  9).  Reflecting  the  overall 
ratings,  SMEs  rated  the  KD  Range  as  being  more  effective  than  ISMT  in  all  six  cases,  and  these 
differences  were  all  statistically  significant.  Also  for  each  of  the  six  categories,  mean  ratings  for  the  KD 
Range  were  above  4,  which  corresponds  to  "Highly  Effective".  Clearly  then,  SMEs  rate  the  KD  Range 
high  in  Effectiveness  in  absolute  terms,  and  as  compared  to  ISMT.  But  again,  even  though  ISMT  was  not 
rated  as  highly,  it  still  fared  reasonably  well.  For  three  of  the  six  categories,  Aiming,  Trigger  Control,  and 
Firing  Positions,  ISMT  scored  above  3  (Moderately  Effective),  and  nearly  so  for  a  fourth  category, 
Weapons  Handling,  with  a  mean  of  2.96.  So  for  these  four  categories,  SME  instructors  felt  that  ISMT 
trained  marksmanship  tasks  in  manner  that  was  close  to  moderately  effective  or  better. 

Looking  more  closely  at  these  four  categories  and  at  ISMT  characteristics,  these  generally  favorable 
results  are  not  surprising.  ISMT  uses  actual  M-16  rifles  that  are  converted  for  simulated  use.  The  sights 
and  trigger  mechanisms  are  the  same  as  those  used  for  qualification,  so  Aiming  and  Trigger  Control  tasks 
can  be  trained  fairly  well.  The  size  and  weight  of  the  rifle  is  the  same,  so  ISMT  should  be  expected  to 
train  Firing  Positions  and  Weapons  Handling  tasks  adequately.  Thus  in  the  opinion  of  the  SMEs,  ISMT  is 
fairly  effective  at  training  tasks  in  these  categories,  but  not  as  effective  as  the  KD  Range.  One  possible 
explanation  as  to  why  ISMT  ratings  fell  short  of  KD  Range  ratings  may  be  related  to  factors  mentioned 
earlier  such  as  decreased  recoil  of  the  ISMT  rifles  and  its  training  effects  on  re-establishing  sight 
alignment/sight  picture.  Another  reason  may  be  the  perceptual  and  visual  system  differences  involved 
in  aiming  at  simulated,  projected  targets  versus  real,  physical  targets. 

For  two  categories,  Zeroing  and  Ballistics,  ISMT  scored  poorly  in  effectiveness  with  mean  ratings  of 
2.23  and  2.16,  respectively.  These  means  were  close  to  a  rating  of  2,  corresponding  to  "Slightly 
Effective".  These  categories  also  had  the  highest  percentage  of  NA  ratings  (see  Figure  8)  with  Zeroing  at 
27.8%  and  Ballistics  at  31.8%.  Taken  together,  the  data  for  these  two  categories  indicate  SMEs  felt  that 
ISMT  either  trained  the  tasks  poorly,  or  that  it  was  not  used  to  train  the  tasks  at  all.  The  response 
patterns  also  indicate  that  there  is  some  disagreement  among  the  SMEs  about  whether  or  not  ISMT  is 
used  to  train  tasks  within  these  categories.  It  may  be  the  case  that  some  instructors  choose  not  to  use 
certain  features  of  ISMT,  such  as  introducing  wind  effects,  if  they  believe  it  is  ineffective  in  training  how 
to  compensate  for  the  wind.  Although  this  reasoning  is  speculative  it  would  help  explain  why  some 
chose  to  rate  certain  items  as  NA  while  others  assigned  low  ratings.  This  is  an  area  that  USMC  training 
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experts  may  want  to  examine  further,  both  for  standardization  purposes,  and  possibly  for  improving  the 
way  certain  features  of  ISMT  are  used. 

In  the  previously  mentioned  companion  study,  a  separate  project  team  has  plans  to  collect  objective 
data  on  KD  Range  and  ISMT  training  Effectiveness.  In  that  effort,  two  groups  of  Marines  will  each 
participate  in  a  different  method  of  training.  One  group  will  train  traditionally  on  the  KD  Range 
exclusively,  while  the  other  group  will  replace  some  KD  Range  sessions  with  ISMT  sessions.  After 
training,  the  marksmanship  qualifying  scores  of  the  two  groups  will  be  compared.  Those  results  will  also 
be  compared  to  the  results  from  the  current  survey  to  see  to  what  degree,  if  any,  our  SME  ratings  are 
predictive  of  qualification  scores. 

By  several  measures,  SMEs  in  the  present  study  rated  the  KD  Range  as  the  more  effective  training 
tool.  In  the  planned  companion  study,  if  the  group  trained  exclusively  on  the  KD  Range  has  reliably 
higher  qualification  scores,  those  results  would  be  an  indicator  that  the  MTS  has  potential  merit  as  a 
predictor  of  training  system  effectiveness.  The  future  project  is  to  be  conducted  on  a  "not-to-interfere" 
basis,  and  extremely  busy  training  schedules  have  not  yet  afforded  the  opportunity  for  data  collection. 
Provided  that  study  can  be  completed  we  will  work  with  that  project  team  to  combine  the  results  of  our 
efforts. 

Correlations 

When  correlation  coefficients  were  calculated  among  the  dimensions  of  Importance,  Difficulty,  ISMT 
Effectiveness,  and  KD  Range  Effectiveness,  the  only  significant  association  was  between  Importance  and 
KD  Range  Effectiveness  (see  Table  3).  Tasks  that  SMEs  viewed  as  high  in  Importance  were  rated  highly  in 
terms  of  KD  Range  Effectiveness  as  well.  So  not  only  did  SMEs  rate  the  KD  Range  as  more  effective,  they 
saw  it  as  particularly  effective  for  more  important  items.  Although  not  quite  significant  at  the  p  =  .05 
level,  there  was  also  a  trend  in  the  opposite  direction  for  the  correlation  between  Difficulty  and  ISMT 
Effectiveness  (r  =  -.41,  p  =  .057),  indicating  that  ISMT  was  viewed  as  less  effective  in  training  the  more 
difficult  tasks.  A  recommendation  emerging  from  these  results  would  be  to  give  KD  Range  priority  to 
tasks  rated  as  more  important  and  difficult.  A  review  of  Tables  1  and  2  shows  that  Aiming  tasks  tended 
to  fall  into  this  category. 

One  other  interesting  correlation  result  was  that  between  task  Importance  and  Difficulty  [r  =  .37,  p  = 
.095).  Although  not  statistically  significant,  there  was  some  tendency  for  items  rated  as  important  to 
also  be  rated  as  difficult.  One  simple  explanation  would  be  that  the  important  items  are  indeed  more 
difficult.  Another  possible  explanation  is  that  the  structure  of  the  survey  may  have  induced  a  confound 
between  the  dimensions  of  Importance  and  Difficulty,  as  follows.  The  most  efficient  way  to  complete 
the  survey  was  to  read  the  task  statement,  and  then  rate  it  in  terms  of  Importance,  Difficulty,  ISMT 
Effectiveness,  and  KD  Range  Effectiveness.  The  fact  that  Importance  was  immediately  followed  by 
Difficulty  may  have  caused  the  former  rating  to  bias  the  latter  in  the  same  direction.  If  follow  on  work 
were  to  be  conducted  it  may  be  worthwhile  to  investigate  new  methods  of  survey  administration,  such 
as  having  the  SMEs  rate  all  items  on  one  dimension  before  moving  on  to  the  next.  More  data  on  this 
topic  may  provide  a  clearer  picture  about  any  relationship  between  Importance  and  Difficulty. 
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CONCLUSIONS 


Several  conclusions  and  recommendations  that  can  be  drawn  from  this  study  are  listed  below. 

•  The  survey  exhibited  favorable  characteristics  as  a  marksmanship  trainer  evaluation  tool,  but  an 
important  future  step  would  be  to  compare  the  subjective  effectiveness  ratings  from  this  survey 
to  objective  training  results,  as  they  become  available. 

•  The  survey  identified  a  clear  SME  preference  for  the  KD  Range  over  ISMT  as  a  training  tool  for 
marksmanship  qualification. 

•  Despite  this  preference,  the  SMEs  still  viewed  ISMT  as  a  useful  training  tool.  Depending  on  how 
the  data  were  analyzed,  ISMT  received  mean  ratings  that  ranged  from  just  below  "Moderately 
Effective"  to  ratings  that  fell  between  "Moderately  Effective"  and  "Highly  Effective". 

•  Based  on  the  response  patterns  and  associations  between  training  effectiveness,  task 
importance,  and  task  difficulty,  the  KD  Range  should  be  used  to  train  items  that  are  more 
important  and  difficult,  such  as  aiming  tasks.  When  KD  Range  time  is  particularly  scarce,  a  good 
use  of  limited  assets  would  be  to  divert  less  difficult  and  less  important  tasks,  such  as  Weapons 
Handling  items,  to  ISMT. 

•  There  was  some  disagreement  among  SMEs  regarding  whether  or  not  ISMT  was  used  to  train 
certain  tasks,  particularly  Ballistics  and  Zeroing  tasks.  It  would  probably  be  useful  for  USMC 
training  experts  and  decision-makers  to  specifically  look  into  which  ISMT  features  are  used  and 
how  they  are  used,  and  conversely,  which  features  are  not  used,  and  why  they  are  not  used. 
Perhaps  some  ISMT  features  are  underutilized,  and/or  simple  improvements  can  be  made  to 
improve  ISMT's  ability  to  train  certain  tasks,  ultimately  easing  the  demand  on  KD  Ranges. 

References 

Allen,  G.A.  &  Rashotte,  M.A.  (2006).  Training  metric  accuracy  in  distance  estimation  skill:  pictures  versus 
words.  Applied  Cognitive  Psychology,  20, 171-186. 

Burke,  M.J.,  Finkelstein,  L.M.,  &  Dusig,  M.S.  (1999).  On  average  deviation  indices  for  estimating 
interrater  agreement.  Organizational  Research  Methods,  2(1),  49-68. 

Dunlap,  W.P.,  Burke,  M.J.,  &  Smith-Crowe.  (2003).  Accurate  tests  of  statistical  significance  for  rwg  and 
average  deviation  interrater  agreement  indexes.  Journal  of  Applied  Psychology,  88(2),  356-362. 

Galesic,  M  &  Bosnjak,  M.  (2009).  Effects  of  questionnaire  length  on  participation  and  indicators  of 
response  quality  in  a  web  survey.  Public  Opinion  Quarterly,  73(2),  349-360. 

Indoor  Simulated  Marksmanship  Trainer  -  Enhanced  (ISMT-E)  (2008).  Retrieved  from 

http://www.marines.mil/unit/pandr/Documents/Concepts/2008/PDF/CP08Ch3P7  Indoor  Simu 

lated  Marksmanship  Trainer%E2%80%93Enhanced.pdf  on  January  6,  2012. 

LeBrenton,  J.M.  &  Senter,  J.L.  (2008).  Answers  to  20  questions  about  interrater  reliability  and  interrater 
agreement.  Organizational  Research  Methods,  11(4),  815-852. 


23 


United  States  Marine  Corps  (2001).  Rifle  Marksmanship.  Retrieved  from 

https://www.google.com/url?q=http://www.everyspec.com/USMC/download.php%3Fspec%3D 

MCRP  3- 

01A  29MAR2001. 013708. pdf&sa=U&ei=  IKrULDGHcmx0AG7ilDQBA&ved=0CA8QFiAE&client=i 

nternal-uds-cse&usg=AFQiCNHCc3tps8PVIBKg4AYI10TQuSWoGAon  May  17,  2010. 

Training  and  Education  Command  (2010).  Concept  of  operations  for  the  Indoor  Simulated 
Marksmanship  Trainer. 

Yates,  W.  (2004).  A  training  transfer  study  of  the  Indoor  Simulated  Marksmanship  Trainer.  Unpublished 
Master's  thesis.  Retrieved  from 

http://www.personal.psu.edu/krml0/Live%20Fire%20Proiect/ISMT%20transfer.pdf  on  January 

6,  2012. 


24 


Appendix  A 


Survey  instructions 

NOTE:  This  survey  is  presented  in  an  Excel  workbook  (see  the  multiple  colored  tabs  below).  Be  sure  to 
read  the  brief  "Informed  Consent"  information  and  the  "Privacy  Act  Statement",  and  to  fill  out  the 
short  "Marksmanship  Experience"  worksheet  as  well.  Please  save  the  file  often  so  you  don't  lose  your 
work.  Finally,  if  you  have  any  questions,  call  Henry  Williams  (937-938-3880),  Cristina  Kirkendall  (937- 
938-3913),  or  Eric  Robinson  (937-938-3919).  They  will  be  happy  to  help  with  any  problems. 


You  will  probably  find  it  helpful  to  print  out  these  instructions  and  rating  scales  and  refer  back  to 
them  as  you  fill  out  the  survey.  You  can  also  see  the  ratings  scales  by  hovering  the  cursor  over  cells 
in  the  top  row  on  the  "Survey"  worksheet. 

On  this  survey,  you  will  see  a  variety  of  tasks  related  to  marksmanship.  We  would  like  you  to  provide 
ratings  for  each  task  statement  using  the  scales  provided  below.  Hovering  the  cursor  over  a  task 
statement  cell  will  give  you  additional  detail  about  that  item.  Please  review  all  task  statements  and 
type  the  number  corresponding  to  your  rating  in  the  appropriate  column.  If  you  think  that  a  task  is  not 
relevant  to  USMC  marksmanship  duties,  select  NA. 

Scales  1  &  2  concern  the  characteristics  of  the  tasks  themselves  while  Scales  3  &  4  relate  to  how  well 
ISMT  and  the  live  fire  range  train  these  tasks.  Please  read  the  definitions  of  the  scales  and  rating 
options  provided  below.  When  providing  ratings  for  the  tasks,  you  may  also  review  the  definitions  and 
ratings  scales  by  hovering  the  cursor  over  the  cell  containing  the  name  of  the  rating  scale  (top  row). 
There  is  also  space  at  the  end  of  each  row  for  any  additional  comments  you  may  have  on  a  task. 


There  are  two  questions  at  the  end  of  the  survey  asking  you  to  rate  ISMT  and  the  Live  Fire  Range  in 
their  general  ability  to  train  marksmanship  tasks.  For  these  items  please  think  about  the  overall 
capabilities  of  both  training  methods  and  choose  the  best  response  for  each  item  from  the  dropdown 
menu. 


Scale  1:  Importance  for  accuracy 

The  degree  to  which  incorrect  performance  of  the  task  would  result  in  reduced  ability  to 
place  rounds  effectively. 

1  Not  important 

2  Slightly  important 

3  Moderately  important 

4  Highly  important 

5  Extremely  important 

NA  Not  applicable 
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Scale  2:  Difficulty  to  Learn 


Difficulty  to  learn  reflects  the  total  amount  of  time  and  effort  required  to  learn  to  perform 
a  task  successfully  and  independently,  relative  to  all  other  marksmanship  training. 

1  One  of  the  easiest  tasks  to  learn 

2  Easier  to  learn  than  most  other  tasks 

^  Approximately  half  of  the  tasks  are  easier  to  learn  and  half  are  more  difficult 
to  learn 

4  Harder  to  learn  than  most  other  tasks 

5  One  of  the  most  difficult  to  learn  of  all  tasks 

NA  Not  applicable 

Scale  3:  ISMT  Effectiveness 

How  effective  is  ISMT  in  training  the  Marine  to  perform  this  task? 

Not  effective  (ISMT  is  used  to  train  this  task,  but  does  not  train  the  task 
effectively) 

2  Slightly  effective 

3  Moderately  effective 

4  Highly  effective 

5  Extremely  effective 

NA  ISMT  is  not  used  to  train  this  task 

Scale  4:  Live  Fire  Effectiveness 

How  effective  is  the  live  fire  range  in  training  the  Marine  to  perform  this  task? 

Not  effective  (The  live  fire  range  is  used  to  train  this  task,  but  does  not  train 
the  task  effectively) 

2  Slightly  effective 

3  Moderately  effective 

4  Highly  effective 

5  Extremely  effective 

NA  The  live  fire  range  is  not  used  to  train  this  task 
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Marksmanship  Experience 


Please  list  the  weapon(s)  which  you  train  others  to  use  (e.g.,  M-16 
rifle,  M9  pistol). 

2  How  long  have  you  been  an  M-16  rifleman?  Please  provide  both  the 

Year(s)  and  Month(s).  Enter  "0"  if  necessary. 

Year(s): 

Month(s): 

3  How  long  have  you  been  training  M-16  marksmanship?  Please 

provide  both  the  Year(s)  and  Month(s).  Enter  "0"  if  necessary. 

Year(s): 

Month(s): 

4  How  long  have  you  worked  with  ISMT?  Please  provide  both  the 

Year(s)  and  Month(s).  Enter  "0"  if  necessary. 

Year(s): 

Month(s): 

5  Other  comments  or  notes  that  you  would  like  to  add: 

Marksmanship  Survey 


Task  Statement 

Importance 

for 

Accuracy 

Difficulty  to 
Learn 

ISMT 

Effectiveness 

Live  Fire 
Range 

Effectiveness 

Additional 

Comments 

(Optional) 

1  Establish  the  correct  sight  alignment. 

2  Establish  the  correct  sight  picture. 

3  Establish  proper  stock  weld. 

4  Establish  proper  eye  relief. 

5  Maintain  sight  alignment  and  sight  picture. 

Fire  at  the  appropriate  point  in  the  breath 
cycle. 

7  Establish  the  proper  trigger  grip. 

Establish  the  proper  trigger  finger 
placement. 

9  After  firing,  reset  the  trigger. 

Maintain  proper  uninterrupted  trigger 
control. 

Maintain  proper  interrupted  trigger 
control. 

12  Apply  marksmanship  fundamentals  until 
the  round  exits  the  barrel  of  the  rifle. 

Bring  the  rifle  sights  back  on  target  for 
additional  shots. 

Select  and  assume  a  firing  position  based 

14  on  mobility,  stability,  and  observation  of 
the  target. 

15  Don  the  loop  sling. 

16  Use  proper  left  hand  position. 

12  Use  proper  placement  of  the  rifle  butt  in 
the  pocket  of  the  shoulder. 
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18  Use  proper  grasp  on  the  pistol  grip. 

19  Use  proper  positioning  of  the  right  elbow. 

Maintain  firm  and  consistent  placement  of 
20  the  cheek  against  the  stock  from  shot  to 
shot. 

Use  skeletal  structure  to  support  rifle's 
weight  when  using  the  loop  sling. 

Relax  the  muscles  when  using  the  loop 
sling. 

23  Adjust  the  body  to  achieve  proper  point  of 
aim  when  using  the  loop  sling. 

24  Assume  the  prone  position 

25  Assume  the  sitting  position 

26  Assume  the  kneeling  position 

27  Assume  the  high  kneeling  position 

28  Assume  the  medium  kneeling  position 

29  Assume  the  low  kneeling  position 

30  Assume  the  standing  position 

Use  the  front  sight  post  to  adjust  for 
elevation. 

^2  Use  the  rear  sight  elevation  knob  to  adjust 
the  sight  for  a  specific  range  to  the  target. 

Use  the  windage  knob  to  adjust  the  strike 
of  the  round  right  or  left. 

Establish  initial  sight  settings  to  serve  as 

34  the  starting  point  for  initial  zeroing  and 
subsequent  adjustments. 

35  Establish  BZO. 

Rezero  in  order  to  compensate  for  the 

36  effects  of  wind,  temperature, 
precipitation,  and  light. 
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Shift  point  of  aim  or  use  offset  aiming 

37  when  conditions  do  not  permit  mechanical 
sight  adjustment. 

If  the  strike  of  the  round  is  known,  aim  the 
next  shot  an  equal  distance  from  center 
mass  in  the  opposite  direction  to 
compensate. 

Use  environmental  features  to  support  the 
weapon. 

Use  unit  of  measure  method  to  determine 
40 

distance  to  the  target. 

Use  front  sight  post  method  to  determine 
distance  to  the  target. 

„  Use  visible  detail  method  to  determine 

42 

distance  to  the  target. 

Use  bracketing  method  to  determine 
distance  to  the  target. 

„„  Use  multiple  methods  to  determine 

44 

distance  to  the  target. 

Use  environmental  cues  to  estimate  wind 
direction,  angle,  and  velocity. 

Use  the  angle  between  a  flag  and  the 
flagpole  in  order  to  estimate  wind  velocity. 

47  Execute  a  controlled  pair  to  a  target. 

Engage  targets  while  wearing  field 
protective  mask. 

Overall,  how  effective  do  you  think  ISMT  is  at  training  marksmanship  tasks  in 

49  general?  Please  click  on  the  cell  to  the  right  and  choose  one  option  from  the 
dropdown  menu. 

Overall,  how  effective  do  you  think  the  Live  Fire  Range  is  at  training  marksmanship 
50  tasks  in  general?  Please  click  on  the  cell  to  the  right  and  choose  one  option  from 
the  dropdown  menu. 
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Appendix  B 


Tertiary  Tasks  by  Category 


Category  Item  # 


Task  Statement 


Firing  14 
positions 

24 

25 

26 

27 

28 

29 

30 


Select  and  assume  a  firing  position  based  on  mobility,  stability,  and  observation  of 
the  target 

Assume  the  prone  position 
Assume  the  sitting  position 
Assume  the  kneeling  position 
Assume  the  high  kneeling  position 
Assume  the  medium  kneeling  position 
Assume  the  low  kneeling  position 
Assume  the  standing  position 


Weapons  15 
handling  16 
17 

19 

20 

21 

22 

23 

39 


Don  the  loop  sling 

Use  proper  left  hand  position 

Use  proper  placement  of  the  rifle  butt  in  the  pocket  of  the  shoulder 
Use  proper  positioning  of  the  right  elbow 

Maintain  firm  and  consistent  placement  of  the  cheek  against  the  stock  from  shot  to 
shot 

Use  skeletal  structure  to  support  the  rifle's  weight  when  using  the  loop  sling 
Relax  the  muscles  when  using  the  loop  sling 

Adjust  the  body  to  achieve  proper  point  of  aim  when  using  the  loop  sling 
Use  environmental  features  to  support  the  weapon 


Aiming  1 
2 

3 

4 

5 

6 

12 

13 

37 

38 

47 

48 


Establish  the  correct  sight  alignment 

Establish  the  correct  sight  picture 

Establish  proper  stock  weld 

Establish  proper  eye  relief 

Maintain  sight  alignment  and  sight  picture 

Fire  at  the  appropriate  point  in  the  breath  cycle 

Apply  marksmanship  fundamentals  until  the  round  exits  the  barrel  of  the  rifle 
Bring  the  rifle  sights  back  on  target  for  additional  shots 

Shift  point  of  aim  or  use  offset  aiming  when  conditions  do  not  permit  mechanical 
sight  adjustment 

If  the  strike  of  the  round  is  known,  aim  the  next  shot  an  equal  distance  from  center 
mass  in  the  opposite  direction  to  compensate 

Execute  a  controlled  pair  to  a  target 

Engage  targets  while  wearing  the  field  protective  mask 


Trigger 

control 

8 


Establish  the  proper  trigger  grip 
Establish  the  proper  trigger  finger  placement 
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Ballistics 


Zeroing 


9  After  firing,  reset  the  trigger 

10  Maintain  proper  uninterrupted  trigger  control 

11  Maintain  proper  interrupted  trigger  control 

18  Use  proper  grasp  on  the  pistol  grip 

40  Use  unit  of  measure  method  to  determine  distance  to  the  target 

41  Use  front  sight  post  method  to  determine  distance  to  the  target 

42  Use  visible  detail  method  to  determine  distance  to  the  target 

43  Use  bracketing  method  to  determine  distance  to  the  target 

44  Use  multiple  methods  to  determine  distance  to  the  target 

31  Use  the  front  sight  post  to  adjust  for  elevation 

32  Use  the  rear  sight  elevation  knob  to  adjust  the  sight  for  a  specific  range  to  the  target 

33  Use  the  windage  knob  to  adjust  the  strike  of  the  round  right  or  left 

34  Establish  initial  sight  settings  to  serve  as  the  starting  point  for  initial  zeroing  and 
subsequent  adjustments 

35  Establish  BZO 

36  Rezero  in  order  to  compensate  for  the  effects  of  wind,  temperature,  precipitation, 
and  light 

45  Use  environmental  cues  to  estimate  wind  direction,  angle,  and  velocity 

46  Use  the  angle  between  a  flag  and  the  flagpole  in  order  to  estimate  wind  velocity 
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Appendix  C 


rwg  and  AD  statistics  for  each  item,  by  category 
Shaded  values  not  significant  at  the  p  =  .05  level 

Difficulty  to  ISMT  KD  Range 

Importance  Learn  Effectiveness  Effectiveness 


Category 

Item 

N 

rwg 

AD 

N 

rwg 

AD 

N 

rwg 

AD 

N 

rwg 

AD 

Weapons 

Don  the  loop  sling. 

22 

0.27 

1.05 

22 

0.47 

0.64 

22 

0.20 

1.09 

22 

0.48 

0.77 

Handling 

Use  proper  left  hand  position. 

22 

0.68 

0.64 

22 

0.49 

0.82 

22 

0.68 

0.45 

22 

0.74 

0.64 

Use  proper  placement  of  the  rifle  butt 
in  the  pocket  of  the  shoulder. 

22 

0.65 

0.68 

22 

0.78 

0.59 

22 

0.58 

0.64 

21 

0.83 

0.59 

Use  proper  positioning  of  the  right 
elbow. 

22 

0.79 

0.50 

22 

0.73 

0.45 

22 

0.66 

0.64 

22 

0.76 

0.45 

Maintain  firm  and  consistent 
placement  of  the  cheek  against  the 
stock  from  shot  to  shot. 

22 

0.64 

0.68 

22 

0.65 

0.59 

22 

0.64 

0.73 

22 

0.79 

0.50 

Use  skeletal  structure  to  support  rifle's 
weight  when  using  the  loop  sling. 

22 

0.72 

0.55 

22 

0.45 

0.77 

22 

0.80 

0.45 

22 

0.79 

0.50 

Relax  the  muscles  when  using  the  loop 
sling. 

22 

0.72 

0.59 

22 

0.49 

0.73 

22 

0.57 

0.59 

22 

0.79 

0.41 

Adjust  the  body  to  achieve  proper 
point  of  aim  when  using  the  loop  sling. 

22 

0.78 

0.59 

22 

0.64 

0.73 

22 

0.61 

0.73 

22 

0.79 

0.50 

Use  environmental  features  to  support 
the  weapon. 

22 

0.58 

0.73 

22 

0.60 

0.68 

14 

0.57 

0.64 

22 

0.64 

0.73 

Firing 

Select  and  assume  a  firing  position 

22 

0.79 

0.50 

22 

0.60 

0.59 

16 

0.28 

1.00 

22 

0.68 

0.45 

Positions 

based  on  mobility,  stability,  and 
observation  of  the  target. 

Assume  the  prone  position 

22 

0.68 

0.55 

22 

0.64 

0.64 

22 

0.15 

1.05 

22 

0.77 

0.45 

Assume  the  sitting  position 

22 

0.73 

0.45 

22 

0.30 

0.91 

22 

0.18 

1.05 

22 

0.73 

0.55 
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Aiming 


Assume  the  kneeling  position 

22 

0.68 

0.55 

Assume  the  high  kneeling  position 

22 

0.69 

0.64 

Assume  the  medium  kneeling  position 

22 

0.60 

0.68 

Assume  the  low  kneeling  position 

22 

0.53 

0.77 

Assume  the  standing  position 

22 

0.14 

1.00 

Establish  the  correct  sight  alignment. 

22 

0.83 

0.18 

Establish  the  correct  sight  picture. 

22 

0.92 

0.18 

Establish  proper  stock  weld. 

22 

0.48 

0.82 

Establish  proper  eye  relief. 

22 

0.77 

0.50 

Maintain  sight  alignment  and  sight 
picture. 

22 

0.85 

0.27 

Shift  point  of  aim  or  use  offset  aiming 
when  conditions  do  not  permit 
mechanical  sight  adjustment. 

22 

0.54 

0.41 

If  the  strike  of  the  round  is  known,  aim 
the  next  shot  an  equal  distance  from 
center  mass  in  the  opposite  direction 
to  compensate. 

22 

0.64 

0.36 

Fire  at  the  appropriate  point  in  the 
breath  cycle. 

22 

0.38 

0.77 

Apply  marksmanship  fundamentals 
until  the  round  exits  the  barrel  of  the 

rifle. 

22 

0.94 

0.14 

Bring  the  rifle  sights  back  on  target  for 
additional  shots. 

22 

0.73 

0.45 

Execute  a  controlled  pair  to  a  target. 

20 

0.76 

0.50 

Engage  targets  while  wearing  field 

18 

0.03 

0.94 

protective  mask. 


22 

0.27 

0.95 

22 

0.19 

0.95 

22 

0.77 

0.50 

22 

0.35 

0.91 

22 

0.15 

1.05 

22 

0.83 

0.41 

22 

0.35 

0.91 

22 

0.22 

1.05 

22 

0.73 

0.59 

22 

-0.07 

1.27 

22 

0.18 

1.05 

22 

0.59 

0.64 

22 

0.20 

0.95 

18 

0.27 

1.06 

22 

0.77 

0.45 

22 

0.30 

1.00 

22 

0.64 

0.55 

22 

0.79 

0.32 

22 

0.67 

0.45 

22 

0.56 

0.64 

22 

0.90 

0.27 

22 

0.57 

0.82 

22 

0.51 

0.77 

22 

0.72 

0.59 

22 

0.72 

0.41 

22 

0.34 

0.86 

22 

0.69 

0.59 

22 

0.71 

0.45 

22 

0.43 

0.64 

22 

0.73 

0.50 

22 

0.32 

0.95 

19 

0.52 

0.84 

22 

0.94 

0.14 

22 

0.38 

0.95 

19 

0.84 

0.32 

22 

0.85 

0.27 

22 

0.45 

0.82 

22 

0.32 

0.95 

22 

0.73 

0.55 

22 

0.71 

0.45 

21 

0.27 

1.00 

22 

0.89 

0.32 

22 

0.75 

0.45 

22 

0.36 

0.86 

22 

0.73 

0.50 

22 

0.45 

0.82 

19 

0.22 

1.00 

22 

0.74 

0.32 

18 

0.50 

0.72 

15 

0.44 

0.93 

18 

0.58 

0.61 
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Trigger 

Establish  the  proper  trigger  grip. 

22 

0.79 

0.41 

22 

0.49 

0.68 

22 

0.39 

0.91 

22 

0.74 

0.55 

Control 

Establish  the  proper  trigger  finger 
placement. 

22 

0.55 

0.77 

22 

0.73 

0.59 

22 

0.38 

0.86 

22 

0.86 

0.32 

After  firing,  reset  the  trigger. 

22 

0.64 

0.64 

22 

0.44 

0.77 

22 

0.44 

0.73 

22 

0.46 

0.68 

Maintain  proper  uninterrupted  trigger 
control. 

22 

0.68 

0.55 

22 

0.60 

0.68 

22 

0.60 

0.59 

22 

0.68 

0.50 

Maintain  proper  interrupted  trigger 
control. 

22 

0.01 

1.18 

22 

0.57 

0.77 

22 

0.54 

0.77 

22 

0.67 

0.68 

Use  proper  grasp  on  the  pistol  grip. 

22 

0.70 

0.68 

22 

0.52 

0.73 

22 

0.68 

0.50 

22 

0.78 

0.59 

Ballistics 

Use  unit  of  measure  method  to 
determine  distance  to  the  target. 

22 

0.69 

0.59 

22 

0.32 

0.95 

15 

0.40 

0.87 

22 

0.62 

0.64 

Use  front  sight  post  method  to 
determine  distance  to  the  target. 

22 

0.49 

0.77 

22 

0.56 

0.77 

15 

0.30 

1.07 

22 

0.49 

0.82 

Use  visible  detail  method  to  determine 
distance  to  the  target. 

22 

0.46 

0.86 

22 

0.74 

0.50 

15 

0.22 

1.07 

22 

0.62 

0.68 

Use  bracketing  method  to  determine 
distance  to  the  target. 

22 

0.56 

0.77 

22 

0.74 

0.55 

15 

0.13 

1.13 

22 

0.67 

0.68 

Use  multiple  methods  to  determine 
distance  to  the  target. 

22 

0.73 

0.50 

22 

0.60 

0.68 

15 

0.01 

1.20 

22 

0.64 

0.64 

Zeroing 

Use  the  front  sight  post  to  adjust  for 
elevation. 

22 

0.78 

0.41 

22 

0.51 

0.77 

16 

0.54 

0.75 

22 

0.68 

0.55 

Use  the  rear  sight  elevation  knob  to 
adjust  the  sight  for  a  specific  range  to 
the  target. 

22 

0.79 

0.32 

22 

0.39 

0.91 

16 

0.65 

0.44 

22 

0.83 

0.41 

Use  the  windage  knob  to  adjust  the 
strike  of  the  round  right  or  left. 

22 

0.79 

0.32 

22 

0.49 

0.82 

17 

0.63 

0.59 

22 

0.89 

0.32 

Establish  initial  sight  settings  to  serve 

22 

0.63 

0.50 

22 

0.32 

0.95 

16 

0.70 

0.50 

20 

0.83 

0.45 

as  the  starting  point  for  initial  zeroing 
and  subsequent  adjustments. 
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Overall 

Scores 


Shaded  values  are  not  significant  at  the  p  =  .05  level. 


22  0.39  0.91 
21  0.50  0.67 


17  0.38  0.94 
17  0.38  0.88 


21  0.87  0.19 

20  0.78  0.57 


22 

0.35 

0.82 

14 

-0.22 

1.14 

22 

0.60 

0.68 

21 

0.42 

1.05 

13 

-0.04 

1.36 

22 

0.68 

0.55 
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Critical  Values  of  the  rwg  and  AD  statistics 


N 

rwg  (a  -  0.05) 

AD  (o  =  0.05) 

3 

>  1.00 

<0.00 

4 

>  1.00 

<0.00 

5 

>0.85 

<0.40 

6 

>0.72 

<0.56 

7 

>0.67 

<0.61 

8 

>0.61 

<0.69 

9 

>0.57 

<0.72 

10 

>0.53 

<0.74 

11 

>0.47 

<0.79 

12 

>0.44 

<0.82 

13 

>0.43 

<0.83 

14 

>0.41 

<0.85 

15 

>0.40 

<0.87 

16 

>0.38 

<0.88 

17 

>0.37 

<0.89 

18 

>0.35 

<0.91 

19 

>0.34 

<0.91 

20 

>0.33 

<0.92 

25 

>0.30 

<0.96 

30 

>0.27 

<0.98 

Critical  values  are  based  on  a  5-category 
rating  scale 


37 


Appendix  D 


Items  Ranked  by  Importance  Rating 


Importance 

Rank 

Item  # 

Task  Statement 

Category 

Mean 

1 

35 

Establish  BZO. 

Zeroing 

4.90 

2 

12 

Apply  marksmanship  fundamentals  until  the 
round  exits  the  barrel  of  the  rifle. 

Aiming 

4.85 

3 

1 

Establish  the  correct  sight  alignment. 

Aiming 

4.80 

4 

2 

Establish  the  correct  sight  picture. 

Aiming 

4.80 

5 

5 

Maintain  sight  alignment  and  sight  picture. 

Aiming 

4.70 

6 

32 

Use  the  rear  sight  elevation  knob  to  adjust  the 
sight  for  a  specific  range  to  the  target. 

Zeroing 

4.65 

7 

33 

Use  the  windage  knob  to  adjust  the  strike  of  the 
round  right  or  left. 

Zeroing 

4.65 

8 

38 

If  the  strike  of  the  round  is  known,  aim  the  next 
shot  an  equal  distance  from  center  mass  in  the 
opposite  direction  to  compensate. 

Aiming 

4.60 

9 

31 

Use  the  front  sight  post  to  adjust  for  elevation. 

Zeroing 

4.55 

10 

37 

Shift  point  of  aim  or  use  offset  aiming  when 
conditions  do  not  permit  mechanical  sight 
adjustment. 

Aiming 

4.55 

11 

13 

Bring  the  rifle  sights  back  on  target  for 
additional  shots. 

Aiming 

4.50 

12 

25 

Assume  the  sitting  position 

Firing  positions 

4.50 

13 

45 

Use  environmental  cues  to  estimate  wind 
direction,  angle,  and  velocity. 

Zeroing 

4.50 

14 

4 

Establish  proper  eye  relief. 

Aiming 

4.45 

15 

10 

Maintain  proper  uninterrupted  trigger  control. 

Trigger  control 

4.45 

16 

34 

Establish  initial  sight  settings  to  serve  as  the 
starting  point  for  initial  zeroing  and  subsequent 
adjustments. 

Zeroing 

4.45 

17 

44 

Use  multiple  methods  to  determine  distance  to 
the  target. 

Ballistics 

4.45 

18 

47 

Execute  a  controlled  pair  to  a  target. 

Aiming 

4.44 

38 


19 

24 

Assume  the  prone  position 

Firing  positions 

4.40 

20 

26 

Assume  the  kneeling  position 

Firing  positions 

4.40 

21 

46 

Use  the  angle  between  a  flag  and  the  flagpole  in 
order  to  estimate  wind  velocity. 

Zeroing 

4.37 

22 

23 

Adjust  the  body  to  achieve  proper  point  of  aim 
when  using  the  loop  sling. 

Weapons  handling 

4.35 

23 

9 

After  firing,  reset  the  trigger. 

Trigger  control 

4.30 

24 

27 

Assume  the  high  kneeling  position 

Firing  positions 

4.30 

25 

36 

Rezero  in  order  to  compensate  for  the  effects  of 
wind,  temperature,  precipitation,  and  light. 

Zeroing 

4.26 

26 

14 

Select  and  assume  a  firing  position  based  on 
mobility,  stability,  and  observation  of  the  target. 

Firing  positions 

4.25 

27 

17 

Use  proper  placement  of  the  rifle  butt  in  the 
pocket  of  the  shoulder. 

Weapons  handling 

4.25 

28 

18 

Use  proper  grasp  on  the  pistol  grip. 

Trigger  control 

4.25 

29 

28 

Assume  the  medium  kneeling  position 

Firing  positions 

4.25 

30 

6 

Fire  at  the  appropriate  point  in  the  breath  cycle. 

Aiming 

4.15 

31 

16 

Use  proper  left  hand  position. 

Weapons  handling 

4.15 

32 

22 

Relax  the  muscles  when  using  the  loop  sling. 

Weapons  handling 

4.15 

33 

29 

Assume  the  low  kneeling  position 

Firing  positions 

4.15 

34 

7 

Establish  the  proper  trigger  grip. 

Trigger  control 

4.05 

35 

20 

Maintain  firm  and  consistent  placement  of  the 
cheek  against  the  stock  from  shot  to  shot. 

Weapons  handling 

4.05 

36 

42 

Use  visible  detail  method  to  determine  distance 
to  the  target. 

Ballistics 

4.05 

37 

43 

Use  bracketing  method  to  determine  distance  to 
the  target. 

Ballistics 

4.05 

38 

21 

Use  skeletal  structure  to  support  rifle's  weight 
when  using  the  loop  sling. 

Weapons  handling 

4.00 

39 

39 

Use  environmental  features  to  support  the 
weapon. 

Weapons  handling 

4.00 

40 

40 

Use  unit  of  measure  method  to  determine 
distance  to  the  target. 

Ballistics 

3.95 

39 


41 

48 

Engage  targets  while  wearing  field  protective 
mask. 

Aiming 

3.94 

42 

30 

Assume  the  standing  position 

Firing  positions 

3.90 

43 

8 

Establish  the  proper  trigger  finger  placement. 

Trigger  control 

3.85 

44 

3 

Establish  proper  stock  weld. 

Aiming 

3.80 

45 

11 

Maintain  proper  interrupted  trigger  control. 

Trigger  control 

3.80 

46 

15 

Don  the  loop  sling. 

Weapons  handling 

3.75 

47 

19 

Use  proper  positioning  of  the  right  elbow. 

Weapons  handling 

3.65 

48 

41 

Use  front  sight  post  method  to  determine 
distance  to  the  target. 

Ballistics 

3.45 

40 


Appendix  E 


Items  Ranked  by  Difficulty  Rating 


Difficulty 

Rank 

Item  # 

Task  Statement 

Category 

Mean 

1 

10 

Maintain  proper  uninterrupted  trigger  control. 

Trigger  control 

4.10 

2 

44 

Use  multiple  methods  to  determine  distance  to  the 
target. 

Ballistics 

4.10 

3 

48 

Engage  targets  while  wearing  field  protective  mask. 

Aiming 

4.06 

4 

41 

Use  front  sight  post  method  to  determine  distance 
to  the  target. 

Ballistics 

3.80 

5 

37 

Shift  point  of  aim  or  use  offset  aiming  when 
conditions  do  not  permit  mechanical  sight 
adjustment. 

Aiming 

3.75 

6 

40 

Use  unit  of  measure  method  to  determine  distance 
to  the  target. 

Ballistics 

3.75 

7 

12 

Apply  marksmanship  fundamentals  until  the  round 
exits  the  barrel  of  the  rifle. 

Aiming 

3.70 

8 

5 

Maintain  sight  alignment  and  sight  picture. 

Aiming 

3.70 

9 

38 

If  the  strike  of  the  round  is  known,  aim  the  next 
shot  an  equal  distance  from  center  mass  in  the 
opposite  direction  to  compensate. 

Aiming 

3.65 

10 

45 

Use  environmental  cues  to  estimate  wind  direction, 
angle,  and  velocity. 

Zeroing 

3.65 

11 

42 

Use  visible  detail  method  to  determine  distance  to 
the  target. 

Ballistics 

3.65 

12 

43 

Use  bracketing  method  to  determine  distance  to 
the  target. 

Ballistics 

3.60 

13 

30 

Assume  the  standing  position 

Firing  positions 

3.50 

14 

46 

Use  the  angle  between  a  flag  and  the  flagpole  in 
order  to  estimate  wind  velocity. 

Zeroing 

3.47 

15 

11 

Maintain  proper  interrupted  trigger  control. 

Trigger  control 

3.35 

16 

23 

Adjust  the  body  to  achieve  proper  point  of  aim 
when  using  the  loop  sling. 

Weapons  handling 

3.30 

17 

6 

Fire  at  the  appropriate  point  in  the  breath  cycle. 

Aiming 

3.30 

18 

35 

Establish  BZO. 

Zeroing 

3.20 

41 


19 

22 

Relax  the  muscles  when  using  the  loop  sling. 

Weapons  handling 

3.05 

20 

2 

Establish  the  correct  sight  picture. 

Aiming 

3.00 

21 

36 

Rezero  in  order  to  compensate  for  the  effects  of 
wind,  temperature,  precipitation,  and  light. 

Zeroing 

3.00 

22 

14 

Select  and  assume  a  firing  position  based  on 
mobility,  stability,  and  observation  of  the  target. 

Firing  positions 

2.95 

23 

21 

Use  skeletal  structure  to  support  rifle's  weight 
when  using  the  loop  sling. 

Weapons  handling 

2.95 

24 

18 

Use  proper  grasp  on  the  pistol  grip. 

Trigger  control 

2.90 

25 

31 

Use  the  front  sight  post  to  adjust  for  elevation. 

Zeroing 

2.80 

26 

29 

Assume  the  low  kneeling  position 

Firing  positions 

2.80 

27 

4 

Establish  proper  eye  relief. 

Aiming 

2.75 

28 

34 

Establish  initial  sight  settings  to  serve  as  the 
starting  point  for  initial  zeroing  and  subsequent 
adjustments. 

Zeroing 

2.75 

29 

13 

Bring  the  rifle  sights  back  on  target  for  additional 
shots. 

Aiming 

2.70 

30 

26 

Assume  the  kneeling  position 

Firing  positions 

2.70 

31 

39 

Use  environmental  features  to  support  the 
weapon. 

Weapons  handling 

2.65 

32 

1 

Establish  the  correct  sight  alignment. 

Aiming 

2.60 

33 

9 

After  firing,  reset  the  trigger. 

Trigger  control 

2.60 

34 

28 

Assume  the  medium  kneeling  position 

Firing  positions 

2.60 

35 

16 

Use  proper  left  hand  position. 

Weapons  handling 

2.55 

36 

27 

Assume  the  high  kneeling  position 

Firing  positions 

2.50 

37 

32 

Use  the  rear  sight  elevation  knob  to  adjust  the  sight 
for  a  specific  range  to  the  target. 

Zeroing 

2.45 

38 

33 

Use  the  windage  knob  to  adjust  the  strike  of  the 
round  right  or  left. 

Zeroing 

2.40 

39 

47 

Execute  a  controlled  pair  to  a  target. 

Aiming 

2.40 

42 


40 

17 

Use  proper  placement  of  the  rifle  butt  in  the  pocket 
of  the  shoulder. 

Weapons  handling 

2.35 

41 

7 

Establish  the  proper  trigger  grip. 

Trigger  control 

2.35 

42 

20 

Maintain  firm  and  consistent  placement  of  the 
cheek  against  the  stock  from  shot  to  shot. 

Weapons  handling 

2.35 

43 

8 

Establish  the  proper  trigger  finger  placement. 

Trigger  control 

2.35 

44 

25 

Assume  the  sitting  position 

Firing  positions 

2.30 

45 

19 

Use  proper  positioning  of  the  right  elbow. 

Weapons  handling 

2.20 

46 

15 

Don  the  loop  sling. 

Weapons  handling 

2.10 

47 

3 

Establish  proper  stock  weld. 

Aiming 

2.05 

48 

24 

Assume  the  prone  position 

Firing  positions 

1.90 

43 


Appendix  F 


Open-ended  comments  from  the  demographics  questionnaire  and  survey 


Item 

Task  Statement 

Comment 

SME 

NA 

(Demographics) 

1  like  to  shoot 

4 

NA 

(Demographics) 

ISMT  is  helpful  and  can  help  anyone  that  uses  this 

5 

NA 

(Demographics) 

We  must  teach;  "perfect  practice  makes  perfect"  and  not  "practice  makes  perfect" 

7 

1 

Establish  the  correct  sight  alignment. 

Lots  of  shooters  make  mistakes  with  this  on  RCO  (Rifle  Combat  Optic) 

16 

Diagrams  for  the  coachs  (sic)  to  keep  on  them  to  show  proper  aiming  would  greatly  help 

22 

2 

Establish  the  correct  sight  picture. 

Once  again  the  diagrams  would  greatly  help 

22 

4 

Establish  proper  eye  relief. 

Another  problem  area  for  many  shooters 

16 

13 

Bring  the  rifle  sights  back  on  target  for  additional  shots. 

ISMT  doesn't  offer  realistic  recoil 

22 

24 

Assume  the  prone  position 

Very  important  in  combat  scenarios 

16 

30 

Assume  the  standing  position 

We  still  can  not  do  the  standing  in  the  ISMT 

22 

48 

Engage  targets  while  wearing  field  protective  mask. 

1  have  never  done  this  so  1  can  not  make  an  informed  decision 

22 

44 


